Skip to content

[DNM][AMD][MI355X] DSv4 agentic-hicache: switch to mlpnopad image, narrow DPA sweep#2297

Closed
seungrokj wants to merge 3 commits into
mainfrom
amd/agentx_dsv4_sgl_mlpopt
Closed

[DNM][AMD][MI355X] DSv4 agentic-hicache: switch to mlpnopad image, narrow DPA sweep#2297
seungrokj wants to merge 3 commits into
mainfrom
amd/agentx_dsv4_sgl_mlpopt

Conversation

@seungrokj

Copy link
Copy Markdown
Collaborator

Summary

  • Switch dsv4-fp4-mi355x-sglang-agentic-hicache image to rocm/pytorch-private:sglang-rocm720-mi35x_dsv4_mlpnopad for MLP no-pad kernel evaluation
  • Disable non-DPA (TP8 conc 1-8), full-DPA (conc 16-64), and HiCache offload sweep arms; keep only DPA no-offload conc [48, 64]
  • Add perf-changelog entry

Test plan

  • Full sweep passes on MI355X cluster with mlpnopad image
  • DPA no-offload conc 48 and 64 produce valid benchmark results

🤖 Generated with Claude Code

…DPA sweep

- Switch dsv4-fp4-mi355x-sglang-agentic-hicache image to
  rocm/pytorch-private:sglang-rocm720-mi35x_dsv4_mlpnopad for MLP no-pad
  kernel evaluation
- Disable non-DPA (TP8 conc 1-8), full-DPA (conc 16-64), and HiCache
  offload arms; keep only DPA no-offload conc [48, 64]
- Add perf-changelog entry

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@seungrokj seungrokj changed the title [AMD][MI355X] DSv4 agentic-hicache: switch to mlpnopad image, narrow DPA sweep [DNM][AMD][MI355X] DSv4 agentic-hicache: switch to mlpnopad image, narrow DPA sweep Jul 21, 2026
@seungrokj seungrokj added AMD full-sweep-enabled agentx AgentX benchmarks, recipes, and infrastructure labels Jul 21, 2026
seungrokj and others added 2 commits July 21, 2026 14:55
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment thread perf-changelog.yaml
Comment on lines +5013 to +5018

- config-keys:
- dsv4-fp4-mi355x-sglang-agentic-hicache
description:
- "Switch image to rocm/pytorch-private:sglang-rocm720-mi35x_dsv4_mlpnopad for MLP no-pad kernel evaluation on DSv4"
- "Disable non-DPA, full-DPA, and HiCache sweep arms; keep only DPA no-offload conc [48, 64]"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 This new perf-changelog entry has two issues that will fail the changelog CI gate: it strips the trailing two spaces from the prior entry's blank separator line (violating AGENTS.md's explicit whitespace-preservation rule and breaking validate_raw_change's byte-for-byte prefix check), and the new entry itself omits the required pr-link field that ChangelogEntry (utils/matrix_logic/validation.py) mandates. Please restore the original separator line untouched and add pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2297 (or the XXX placeholder) to the new entry.

Extended reasoning...

This PR's perf-changelog.yaml addition has two concrete, verifiable defects that will fail the changelog validation gate.

1. Whitespace on the separator line was altered. AGENTS.md is explicit: "Never delete or modify whitespace in perf-changelog.yaml - CI depends on exact whitespace (including trailing spaces on blank separator lines). Altering it breaks CI." I diffed the actual git bytes: HEAD~1:perf-changelog.yaml ends with ...pull/2258\n \n — i.e. the blank line separating entries carries two trailing spaces. The PR head replaces that with a bare ...pull/2258\n\n- config-keys:..., stripping the two trailing spaces. This is enforced, not just documented: utils/validate_perf_changelog.py's validate_raw_change() requires head_raw.startswith(base_raw) for the append-only flow, and since the byte immediately after 2258\n changed from a space to the next line's -, that startswith check fails, raising "appended entries changed historical perf-changelog.yaml bytes; restore the base file byte-for-byte and append at the end." This function is invoked from utils/prepare_perf_changelog_merge.py, so it fails at merge-preparation time.

2. The new entry is missing the required pr-link field. ChangelogEntry in utils/matrix_logic/validation.py declares pr_link: str = Field(alias="pr-link") with no default, under model_config = ConfigDict(extra="forbid", populate_by_name=True) — making pr-link a required field with no fallback. The entry appended by this PR (config-keys: dsv4-fp4-mi355x-sglang-agentic-hicache) contains only config-keys and description, no pr-link. Both utils/process_changelog.py (line ~159, called to build the sweep matrix) and utils/validate_perf_changelog.py's parse_changelog call ChangelogEntry.model_validate(...) on every added/parsed entry, so the missing required field raises a Pydantic validation error and aborts before the sweep matrix or changelog validation can complete. Every other entry in the file — 626 of them — carries a pr-link, confirming this is a hard convention, not an optional field that happens to usually be filled in.

Why nothing else catches this before CI: the pr-link placeholder allowance (XXX or the PR's own URL, via PR_LINK_PLACEHOLDERS) only kicks in after model_validate succeeds — omitting the key entirely fails hard before that check ever runs. Likewise the whitespace rule is easy to trip accidentally: any editor/tool that trims trailing whitespace on save, or a manual copy-paste of the tail of the file, will silently drop the two trailing spaces on the separator line, which is exactly what happened here.

Concrete proof:

$ git show HEAD~1:perf-changelog.yaml | tail -c 20 | cat -A
github.com/SemiAnalysisAI/InferenceX/pull/2258$
  $                                    <- blank line, TWO trailing spaces + $
$ tail -c 300 perf-changelog.yaml | cat -A
2258$
$                                       <- blank line, NO trailing spaces (bug 1)
- config-keys:$
    - dsv4-fp4-mi355x-sglang-agentic-hicache$
  description:
    - "..."
    - "..."
                                        <- entry ends here; no pr-link key at all (bug 2)

Feeding this head content through validate_raw_change(base_raw, head_raw) returns head_raw.startswith(base_raw) == False (first divergent byte right after 2258\n), and feeding the new entry dict through ChangelogEntry.model_validate() raises pr_link\n Field required.

Fix: keep the original pull/2258\n \n line byte-for-byte, and append the new entry directly after it with a pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2297 line (or the XXX placeholder used by other in-flight PRs before their PR number is known) added to the new entry's fields.

Comment thread configs/amd-master.yaml

dsv4-fp4-mi355x-sglang-agentic-hicache:
image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
image: rocm/pytorch-private:sglang-rocm720-mi35x_dsv4_mlpnopad

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This PR retargets dsv4-fp4-mi355x-sglang-agentic-hicache to rocm/pytorch-private:sglang-rocm720-mi35x_dsv4_mlpnopad, a mutable, non-digest tag intended for iterative kernel evaluation. Because runners/launch_mi355x-amds.sh caches the enroot squashfs keyed only on the image string and skips re-import whenever that file already exists, any runner that has already pulled this tag once will silently keep serving the stale build on every future re-push, with no error — the exact caching hazard already documented on the neighboring dsv4-fp4-mi355x-vllm entry in this file. Consider pinning to a digest-suffixed tag (e.g. @sha256:...) to avoid silently stale benchmark results during the mlpnopad iteration.

Extended reasoning...

The mechanism: runners/launch_mi355x-amds.sh (~lines 264-286) derives SQUASH_FILE deterministically from the IMAGE string alone (sed 's/[\/:@#]/_/g') and stores it at a persistent, node-local path under /var/lib/squash/. The import logic is:

if unsquashfs -l "$SQUASH_FILE" > /dev/null 2>&1; then
  echo "Squash file already exists and is valid, skipping import"
else
  rm -f "$SQUASH_FILE"
  enroot import ... docker://$IMAGE
fi

There is no comparison against the remote registry digest and no TTL/expiry on the cached file — once a squashfs exists for a given image-string key on a runner, it is reused unconditionally on every subsequent job that references the same tag.

Why this PR triggers it: configs/amd-master.yaml:1539 switches dsv4-fp4-mi355x-sglang-agentic-hicache from the previous tag lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 (date-stamped, effectively immutable) to rocm/pytorch-private:sglang-rocm720-mi35x_dsv4_mlpnopad. Per the PR description, this new tag exists specifically for iterative MLP no-pad kernel evaluation — i.e. the vendor is expected to push multiple updated builds under this exact same tag string while tuning the kernel. That is precisely the scenario the launcher's caching logic cannot handle safely: the tag string never changes, so SQUASH_FILE's key never changes, so the cache is never invalidated.

Why nothing in the existing code prevents this: the import guard only checks local squashfs validity (unsquashfs -l succeeding), which tells you the cached file is a well-formed image — not that it matches what's currently at the registry. There's no digest check, no docker manifest inspect-style comparison, and no forced re-pull path exposed through this config.

Impact: on any mi355x-amds runner that has already executed this config once, a subsequent vendor push of an updated mlpnopad build to the same tag is silently ignored — the sweep runs against the old kernel and reports benchmark numbers that do not reflect the intended change, with no error, warning, or log signal distinguishing it from a correct run. This is especially costly in an iterative-evaluation workflow where the whole point is to compare kernel revisions run-over-run.

Proof (step-by-step):

  1. Vendor pushes build A to rocm/pytorch-private:sglang-rocm720-mi35x_dsv4_mlpnopad.
  2. CI runs this PR's sweep on runner X. SQUASH_FILE doesn't exist yet -> enroot import pulls build A -> squashfs cached at /var/lib/squash/rocm_pytorch-private_sglang-rocm720-mi35x_dsv4_mlpnopad.sqsh.
  3. Vendor iterates on the mlpnopad kernel and re-pushes build B to the same tag.
  4. A later sweep is scheduled on the same runner X (same tag string in the config). SQUASH_FILE computed from the image string is identical to step 2, so unsquashfs -l succeeds against the cached (build A) file, prints 'Squash file already exists and is valid, skipping import', and the job runs build A again — not build B.
  5. Benchmark results are recorded as if they reflect build B's kernel changes, but they were actually produced by the stale build A, with no indication to the author or reviewer that this happened.

Precedent: this exact caching hazard is already called out verbatim as an IMPORTANT comment on the neighboring dsv4-fp4-mi355x-vllm entry in this same file: 'pin to a digest-suffixed nightly tag rather than the floating :nightly... the floating tag silently keeps a stale build even after Docker Hub updates.' The new tag in this PR reintroduces the same pattern that comment warns against.

Fix: pin the image to a digest-suffixed reference (rocm/pytorch-private@sha256:<digest>) instead of the mutable tag, so each new build gets a distinct cache key and is guaranteed to be freshly pulled.

Severity note: this does not break the PR's own test plan — the first CI run on a fresh runner pulls the current image correctly, so the stated 'DPA no-offload conc 48/64' validation will pass. The risk only materializes on a later re-push to the same tag on a runner that already cached it, which is a real but not immediate/blocking concern for an internal evaluation config. Filing as a nit/suggestion to pin by digest, consistent with the precedent already documented elsewhere in this file.

Comment thread configs/amd-master.yaml

dsv4-fp4-mi355x-sglang-agentic-hicache:
image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
image: rocm/pytorch-private:sglang-rocm720-mi35x_dsv4_mlpnopad

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This PR's title and description are English-only, but AGENTS.md mandates bilingual PR titles/descriptions (format: <English title> / <中文标题>, plus a ## 中文说明 section mirroring the summary) for every PR. Please retitle and add the Chinese translation before merging — recent PRs like #2231 and #2232 follow this convention.

Extended reasoning...

AGENTS.md line 7 is explicit and unconditional: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary; don't translate code blocks, logs, or stack traces — summarize around them)... This applies to every PR and every issue."

This PR's title, [AMD][MI355X] DSv4 agentic-hicache: switch to mlpnopad image, narrow DPA sweep, has no / <中文标题> suffix, and the body (Summary + Test plan) has no ## 中文说明 section translating the change. This isn't a subjective style nit invented by the reviewer — it's a literal, actively-enforced repo convention. The git log shows immediately preceding merged PRs following it correctly: 01e9e73 ... / 将 LMCache 分支并入官方 DSV4 FP4 B200 vLLM AgentX 扫描 (#2231) and 9ecc56d ... / 在调优后的 DSV4 FP4 B300 vLLM AgentX 配方上新增 LMCache 分支 (#2232).

Proof / concrete example: Compare this PR's title against the format the rule requires:

Nothing in the diff itself is wrong — the YAML changes to configs/amd-master.yaml (image swap, sweep-arm pruning) and the perf-changelog.yaml entry are functionally sound and unaffected by this. This is filed against configs/amd-master.yaml only because the bug tracker requires a diff-file location; the actual defect is in the PR title/description metadata, not in any line of code.

Why this doesn't block merge: there's no CI gate enforcing the bilingual-title/description rule (unlike the CODEOWNER sign-off phrase, which does have an automated checker per the recipe-reminder bot comment). Several other recently-merged PRs visible in the log (e.g. #2258 "vLLM DeepSeek-V4 B300 aggregate MTP", #2288 "CollectiveX: add UCCL-EP...") also ship English-only titles, so non-compliance clearly doesn't block merging in practice. The fix is trivial — retitle to append / <中文标题> and add a ## 中文说明 section — but it's a process/documentation gap, not a functional defect, so it's best treated as a nit for the author to clean up rather than a merge-blocking issue.

@github-actions

Copy link
Copy Markdown
Contributor

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agentx AgentX benchmarks, recipes, and infrastructure AMD full-sweep-enabled

Projects

Development

Successfully merging this pull request may close these issues.

1 participant